Robust Graph Representation Learning via Predictive Coding

Predictive coding is a message-passing framework initially developed to model information processing in the brain, and now also topic of research in machine learning due to some interesting properties. One of such properties is the natural ability of generative models to learn robust representations thanks to their peculiar credit assignment rule, that allows neural activities to converge to a solution before updating the synaptic weights. Graph neural networks are also message-passing models, which have recently shown outstanding results in diverse types of tasks in machine learning, providing interdisciplinary state-of-the-art performance on structured data. However, they are vulnerable to imperceptible adversarial attacks, and unfit for out-of-distribution generalization. In this work, we address this by building models that have the same structure of popular graph neural network architectures, but rely on the message-passing rule of predictive coding. Through an extensive set of experiments, we show that the proposed models are (i) comparable to standard ones in terms of performance in both inductive and transductive tasks, (ii) better calibrated, and (iii) robust against multiple kinds of adversarial attacks.Comment: 27 Pages, 31 Figure

BoxE: A Box Embedding Model for Knowledge Base Completion

Knowledge base completion (KBC) aims to automatically infer missing facts by exploiting information already present in a knowledge base (KB). A promising approach for KBC is to embed knowledge into latent spaces and make predictions from learned embeddings. However, existing embedding models are subject to at least one of the following limitations: (1) theoretical inexpressivity, (2) lack of support for prominent inference patterns (e.g., hierarchies), (3) lack of support for KBC over higher-arity relations, and (4) lack of support for incorporating logical rules. Here, we propose a spatio-translational embedding model, called BoxE, that simultaneously addresses all these limitations. BoxE embeds entities as points, and relations as a set of hyper-rectangles (or boxes), which spatially characterize basic logical properties. This seemingly simple abstraction yields a fully expressive model offering a natural encoding for many desired logical properties. BoxE can both capture and inject rules from rich classes of rule languages, going well beyond individual inference patterns. By design, BoxE naturally applies to higher-arity KBs. We conduct a detailed experimental analysis, and show that BoxE achieves state-of-the-art performance, both on benchmark knowledge graphs and on more general KBs, and we empirically show the power of integrating logical rules.Comment: Proceedings of the Thirty-Fourth Annual Conference on Advances in Neural Information Processing Systems (NeurIPS 2020). Code and data available at: http://www.github.com/ralphabb/Box

Predictive Coding Can Do Exact Backpropagation on Any Neural Network

Intersecting neuroscience and deep learning has brought benefits and developments to both fields for several decades, which help to both understand how learning works in the brain, and to achieve the state-of-the-art performances in different AI benchmarks. Backpropagation (BP) is the most widely adopted method for the training of artificial neural networks, which, however, is often criticized for its biological implausibility (e.g., lack of local update rules for the parameters). Therefore, biologically plausible learning methods (e.g., inference learning (IL)) that rely on predictive coding (a framework for describing information processing in the brain) are increasingly studied. Recent works prove that IL can approximate BP up to a certain margin on multilayer perceptrons (MLPs), and asymptotically on any other complex model, and that zero-divergence inference learning (Z-IL), a variant of IL, is able to exactly implement BP on MLPs. However, the recent literature shows also that there is no biologically plausible method yet that can exactly replicate the weight update of BP on complex models. To fill this gap, in this paper, we generalize (IL and) Z-IL by directly defining them on computational graphs. To our knowledge, this is the first biologically plausible algorithm that is shown to be equivalent to BP in the way of updating parameters on any neural network, and it is thus a great breakthrough for the interdisciplinary research of neuroscience and deep learning.Comment: 15 pages, 9 figure

Bird-Eye Transformers for Text Generation Models

Transformers have become an indispensable module for text generation models since their great success in machine translation. Previous works attribute the~success of transformers to the query-key-value dot-product attention, which provides a robust inductive bias by the fully connected token graphs. However, we found that self-attention has a severe limitation. When predicting the (i+1)-th token, self-attention only takes the i-th token as an information collector, and it tends to give a high attention weight to those tokens similar to itself. Therefore, most of the historical information that occurred before the i-th token is not taken into consideration. Based on this observation, in this paper, we propose a new architecture, called bird-eye transformer(BET), which goes one step further to improve the performance of transformers by reweighting self-attention to encourage it to focus more on important historical information. We have conducted experiments on multiple text generation tasks, including machine translation (2 datasets) and language models (3 datasets). These experimental~results show that our proposed model achieves a better performance than the baseline transformer architectures on~all~datasets. The code is released at: \url{https://sites.google.com/view/bet-transformer/home}

Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning

How the brain performs credit assignment is a fundamental unsolved problem in neuroscience. Many `biologically plausible' algorithms have been proposed, which compute gradients that approximate those computed by backpropagation (BP), and which operate in ways that more closely satisfy the constraints imposed by neural circuitry. Many such algorithms utilize the framework of energy-based models (EBMs), in which all free variables in the model are optimized to minimize a global energy function. However, in the literature, these algorithms exist in isolation and no unified theory exists linking them together. Here, we provide a comprehensive theory of the conditions under which EBMs can approximate BP, which lets us unify many of the BP approximation results in the literature (namely, predictive coding, equilibrium propagation, and contrastive Hebbian learning) and demonstrate that their approximation to BP arises from a simple and general mathematical property of EBMs at free-phase equilibrium. This property can then be exploited in different ways with different energy functions, and these specific choices yield a family of BP-approximating algorithms, which both includes the known results in the literature and can be used to derive new ones.Comment: 31/05/22 initial upload; 22/06/22 change corresponding author; 03/08/22 revision

Predictive Coding beyond Gaussian Distributions

A large amount of recent research has the far-reaching goal of finding training methods for deep neural networks that can serve as alternatives to backpropagation (BP). A prominent example is predictive coding (PC), which is a neuroscience-inspired method that performs inference on hierarchical Gaussian generative models. These methods, however, fail to keep up with modern neural networks, as they are unable to replicate the dynamics of complex layers and activation functions. In this work, we solve this problem by generalizing PC to arbitrary probability distributions, enabling the training of architectures, such as transformers, that are hard to approximate with only Gaussian assumptions. We perform three experimental analyses. First, we study the gap between our method and the standard formulation of PC on multiple toy examples. Second, we test the reconstruction quality on variational autoencoders, where our method reaches the same reconstruction quality as BP. Third, we show that our method allows us to train transformer networks and achieve a performance comparable with BP on conditional language models. More broadly, this method allows neuroscience-inspired learning to be applied to multiple domains, since the internal distributions can be flexibly adapted to the data, tasks, and architectures used

Brain-Inspired Computational Intelligence via Predictive Coding

Artificial intelligence (AI) is rapidly becoming one of the key technologies of this century. The majority of results in AI thus far have been achieved using deep neural networks trained with the error backpropagation learning algorithm. However, the ubiquitous adoption of this approach has highlighted some important limitations such as substantial computational cost, difficulty in quantifying uncertainty, lack of robustness, unreliability, and biological implausibility. It is possible that addressing these limitations may require schemes that are inspired and guided by neuroscience theories. One such theory, called predictive coding (PC), has shown promising performance in machine intelligence tasks, exhibiting exciting properties that make it potentially valuable for the machine learning community: PC can model information processing in different brain areas, can be used in cognitive control and robotics, and has a solid mathematical grounding in variational inference, offering a powerful inversion scheme for a specific class of continuous-state generative models. With the hope of foregrounding research in this direction, we survey the literature that has contributed to this perspective, highlighting the many ways that PC might play a role in the future of machine learning and computational intelligence at large.Comment: 37 Pages, 9 Figure

Roma: Oltre le baraccopoli: Agenda politica per ripartire dalle periferie dimenticate

Con il presente documento, presentato in vista delle elezioni comunali che si svolgeranno a Roma nel 2016, l’Associazione 21 luglio vuole proporre alle forze politiche e ai candidati a cariche elettive i principi essenziali per mutare radicalmente le politiche verso gli abitanti delle baraccopoli e dei micro insediamenti presenti nella Capitale. Le azioni previste nel documento hanno come obiettivo, nell’arco temporale di 5 anni: la chiusura graduale e progressiva delle baraccopoli e dei micro insediamenti della Capitale e il superamento dei centri di raccolta dove sono concentrate le famiglie vittime degli sgomberi che nel passato hanno coinvolto abitanti di numerose baraccopoli. “Roma: oltre le baraccopoli” si avvale degli studi condotti dall’Associazione 21 luglio e, nell’ultima parte, del prezioso apporto del prof. Tommaso Vitale, Sciences Po (Université Sorbonne Paris Cité)1. Il testo condivide medesimi principi e metodi riportati all’interno della “Delibera di iniziativa popolare per il superamento dei campi rom”, promossa da nove associazioni2 e sottoscritta da oltre 6.000 cittadini, depositata in Campidoglio l’11 settembre 2015

Effect of dietary supplementation with ultramicronized palmitoylethanolamide in maintaining remission in cats with nonflea hypersensitivity dermatitis: a double-blind, multicentre, randomized, placebo-controlled study

Background Feline nonflea hypersensitivity dermatitis (NFHD) is a frequent cause of over-grooming, scratching and skin lesions. Multimodal therapy often is necessary. Hypothesis/Objectives To investigate the efficacy of ultramicronized palmitoylethanolamide (PEA-um) in maintaining methylprednisolone-induced remission in NFHD cats. Animals Fifty-seven NFHD cats with nonseasonal pruritus were enrolled originally, of which 25 completed all study requirements to be eligible for analysis. Methods and materials Cats were randomly assigned to PEA-um (15 mg/kg per os, once daily; n = 29) or placebo (n = 28) while receiving a 28 day tapering methylprednisolone course. Cats responding favourably to methylprednisolone were then administered only PEA-um (n = 21) or placebo (n = 23) for another eight weeks, followed by a four week long treatment-free period. Cats were maintained in the study until relapse or study end, whichever came first. Primary outcome was time to relapse. Secondary outcomes were pruritus Visual Analog Scale (pVAS), SCORing Feline Allergic Dermatitis scale (SCORFAD) and owner Global Assessment Score (GAS). Results Mean relapse time was 40.5 days (+/- 7.8 SE) in PEA-um treated cats (n = 13) and 22.2 days (+/- 3.7 SE) for placebo (n = 12; P = 0.04). On Day 28, the severity of pruritus was lower in the PEA-um treated cats compared to placebo (P = 0.03). Mean worsening of pruritus at the final study day was lower in the PEA-um group compared to placebo (P = 0.04), whereas SCORFAD was not different between groups. Mean owner GAS at the final study day was better in the PEA-um than the placebo-treated group (P = 0.05). Conclusion and clinical importance Ultramicronized palmitoylethanolamide could represent an effective and safe option to delay relapse in NFHD cats